System, computer program and method for 3D object measurement, modeling and mapping from single imagery

ABSTRACT

Methods, computer programs, and computer systems facilitate deriving three-dimensional measurement information and/or creating three-dimensional models and maps, from single images of three-dimensional objects. Aspects of the invention include obtaining at least one two-dimensional single image of the object, deriving three-dimensional coordinate information associated with the image, and obtaining three-dimensional measurements based on projection and/or shadow measurements of the object and metadata derived from the single image. In another aspect of the method, the method includes the further step of creating three-dimensional models or maps based on the projection and/or shadow measurements.

FIELD OF THE INVENTION

The present invention relates to mapping, surveying, photogrammetry,remote sensing, visualization and simulation, gaming, planning,geomatics, civil engineering, geography, and more particularly relatesto the collection of measurement and dimension information of or betweenobjects from single images, and the three-dimensional (3D) models andmaps and the subsequent use of such information in analysis, modeling,mapping and visualization.

BACKGROUND OF THE INVENTION

A conventional method of measuring 3D objects is called stereo-visionthat obtains paired stereo images taken of the same objects. When theimagery geometry model (IGM) of each image is given, the 3D coordinateinformation of the objects can be determined. A photogrammetric methodsurveys the objects by selecting conjugate points, and thereby measureany dimension using the IGMs based on these points. The 3D models andmaps are then generated by using the stereo-vision approaches.

The acquisition of stereo images especially from airborne or satellitesensors is more expensive and needs a longer delivery time compared withacquiring single images. Also the majority of the archived images in thedatabases maintained by imagery vendors are single images. Therefore,the use of single images has advantages for applications such asemergency mapping, defense, intelligence, telecommunication andengineering etc.

There is no known system that has been developed to perform 3Dmeasurement, modeling and mapping from the single images. The presentinvention has resulted in an operational method, computer program andsystem that can effectively obtain 3D measurements and create 3D objectmodels and maps. The system is comprised of unique utilities and novelalgorithms that are designed to make use of object projection, shadow,object geometry, and the IGM.

The IGM describes the geometric relationship between the object spaceand the image space, or vice visa. The two broadly used IGMs include thephysical sensor model and the generalized sensor model. The rationalfunction model (RFM) is a kind of generalized sensor model.

The following relevant prior art has been identified:

Jiang, W., Tao, C. V., Hu, Y., Xu, Z., 2003. 3-D measurement from singleand stereo high-resolution satellite imagery based on the RFM, ASPRSAnnual Conference, 5-9 May, Anchorage, Ak., 7 p. (This referencedescribes several experimental results obtained using RFM based methodsfrom the satellite imagery)OpenGIS Consortium, 1999. The OpenGIS Abstract Specification—Topic 7.The Earth Imagery Case. (This reference provides an overview of IGMsused in the mapping, remote sensing and geospatial industry).Tao, C. V., Hu, Y., 2001. A comprehensive study of the rational functionmodel for photogrammetric processing, Photogrammetric Engineering &Remote Sensing, 67(12): 1347-1357. (This reference provides a detailedmathematical formulation of the RFM sensor model and its experimentalstudy on its accuracy).

SUMMARY OF THE INVENTION

In accordance with one aspect of the present invention, there isprovided a method for deriving three-dimensional measurement informationand/or creating three-dimensional models and maps, from single images ofat least one three-dimensional object, the method comprising the stepsof: obtaining at least one two-dimensional single image of the object,the image consisting of image data and being associated with an imagegeometry model (IGM); deriving three-dimensional coordinate informationassociated with the image, based on the IGM, and associating thethree-dimensional coordinate information with the image data; analyzingthe image data so as to measure the projection of the object using theIGM to derive measurement data including the height and/orpoint-to-point distances pertaining to the object; and/or, measure theshadow of the object to derive measurement data including the heightand/or point-to-point distance pertaining to the object; and obtainingthree-dimensional measurements based on the projection and/or shadowmeasurements of the object.

In accordance with another aspect of the present invention, there isprovided a system for deriving three-dimensional measurement informationand/or creating three-dimensional models and maps, from single images ofat least one three-dimensional object, the system comprising: at leastone computer; and a computer program operably linked to the computer soas to enable the computer to: obtain at least one two-dimensional singleimage of the object, the image consisting of image data and beingassociated with an image geometry model (IGM); derive three-dimensionalcoordinate information associated with the image, based on the IGM, andassociating the three-dimensional coordinate information with the imagedata; analyze the image data so as to: measure the projection of theobject using the IGM to derive measurement data including the heightand/or point-to-point distances pertaining to the object; and/or measurethe shadow of the object to derive measurement data including the heightand/or point-to-point distance pertaining to the object; and obtainthree-dimensional measurements based on the projection and/or shadowmeasurements of the object.

In accordance with a further aspect of the present invention, there isprovided a computer usable medium, the computer useable mediumcomprising instructions for defining a measurement utility on acomputer, the measurement utility being operable to: obtain at least onetwo-dimensional single image of the object, the image consisting ofimage data and being associated with an image geometry model (IGM);derive three-dimensional coordinate information associated with theimage, based on the IGM, and associating the three-dimensionalcoordinate information with the image data; analyze the image data so asto: measure the projection of the object using the IGM to derivemeasurement data including the height and/or point-to-point distancespertaining to the object; and/or measure the shadow of the object toderive measurement data including the height and/or point-to-pointdistance pertaining to the object; and obtain three-dimensionalmeasurements based on the projection and/or shadow measurements of theobject.

The present invention provides a method for deriving 3D measurementinformation and creating 3D models and maps from the single imagery withthe IGM support, including the RFM sensor model, where: (i) a singleimage is available, (ii) both static and dynamic objects are to bemeasured, (iii) no stereo viewing devices are available, or (iv)conjugate points from stereo image pairs cannot be identified.

The present invention includes a system that enables measurementinformation to be readily accessed, and used from an image in accordancewith the method of the present invention. This system consists ofutilities to obtain 3D measurements and to create 3D models and 3D mapsfrom the single image. The system consists generally of a computersystem which is adapted to process instructions provided by the computerprogram of the present invention.

The computer program of the present invention consists of a computerapplication that includes (1) a measurement utility that enablesmeasurements to be made from a 2D image that supports the IGM, themeasurement utility including a projection measurement utility and ashadow measurement utility. The projection measurement utility and theshadow measurement utility co-operate so as to enable point-to-pointmeasurements by operation of the projection measurement utility, theshadow measurement utility, or both, with the support of the IGM; and(2) a model generation utility that efficiently create 3D models andmaps based on measurements made by the measurement utility of theinvention. With the RFM, the application can be applied for any images(e.g., satellite or aerial) supporting the RFM or the like. The presentinvention enables the 3D measurements, models and maps from the singleimages.

Another aspect of the method of the present invention consists of amethod for obtaining measurement information, namely, the distancebetween any two points in the three dimensions from a single image, andfor creating 3D models from the measurements and subsequently generating3D maps from a single image. Specifically, the present inventionprovides methods for:

-   1. Measuring the projection of an object or objects by operation of    the projection measurement utility using the IGM to derive the    heights, point-to-point distances of objects and the like;-   2. Measuring the shadow of an object or objects by operation of the    shadow measurement utility using IGM to derive the heights,    point-to-point distance of both static (including buildings,    overpasses, bridges, etc.) and dynamic objects (including airplanes    in the air). These objects may not have footprints on the ground.-   3. Obtaining the 3D measurements by using the cooperation of    projection and/or shadow data with the IGM.-   4. Creating 3D models and maps by using model generation utility    that implements one or more algorithms.

Another aspect of the method of the present invention is the applicationof specific algorithms of the present invention in order to take themeasurements described.

The present invention can be applied for any images with the RFMsupport. This is due to the fact that the RFM is sensor independent andcan be universally applied to multiple sensors. Thus the computerprogram resulted from this invention can be used for any images withouta need to change its underlying sensor model.

BRIEF DESCRIPTION OF THE DRAWINGS

A detailed description of the certain embodiments of the invention isprovided herein below by way of example only and with reference to thefollowing drawings, in which:

FIG. 1 a: A conceptual drawing of a representative graphic userinterface for accessing the functions of the computer program product ofthe present invention. The toolbar buttons illustrated enable access tothe functions of the utilities of the present invention.

FIG. 1 b: A program resource diagram illustrating the resources of thecomputer program of the present invention.

FIG. 2 a: The conceptual drawing of the measurement utility 8 displayedin the image display window.

FIG. 2 b: A schematic diagram showing the relationship among theprojection and the shadow cast in the image, the IGM, the object, andthe sun angles, assuming a flat ground surface.

FIG. 3: Illustration of determining the horizontal position (X, Y) byintersecting a plane having the elevation of Z in a single image. Theelevation Z is adjusted at some incremental change.

FIG. 4 a: A schematic diagram of the height measurement of a buildingusing the projection measurement utility based on the IGM.

FIG. 4 b: Illustration of the height measurement of a building using theprojection measurement utility of the present invention based on theIGM.

FIG. 5: A diagram showing the relationship between the sun's positionand the displacement of an object's shadow.

FIG. 6: A schematic diagram illustrating the method of the presentinvention, and more specifically showing a method of drawing theprojection and the shadow of an object, assuming a non-flat groundsurface.

FIG. 7 a: A block diagram illustrating the present invention, showingthe measurement of the object height using the shadow information, formeasurable shadows in the image.

FIG. 7 b: A block diagram illustrating the present invention, showingthe measurement of the object height using the shadow information, forun-measurable shadows in the image.

FIG. 8 a: A schematic diagram showing the taking of a height measurementusing the shadow measuring utility for an object that is a building,starting from its base point.

FIG. 8 b: Illustration of the present invention showing the taking of aheight measurement using the shadow measuring utility for an object thatis a building, starting from its base point.

FIGS. 9 a, 9 b, 9 c and 9 d: Illustrations of height measurement inaccordance with the present invention using the shadow measurementutility for a) an airplane in the case of FIG. 9 a, b) an overpass inthe case of FIG. 9 b, c) a tree in the case of FIG. 9 c, d) a chimney inthe case of FIG. 9 d, starting from their shadow endpoints.

FIG. 10 a: Illustration of determination of the base point using theshadow measurement utility of the present invention, where the height ofthe airplane is shown to be 57.1 m.

FIG. 10 b: Illustration of drawing a line mark in accordance with thepresent invention, the line connecting the base point with the landingpoint of the runway, and the horizontal distance being shown as being158.1 m.

FIG. 10 c: Illustrating the present invention by raising the base pointof the line mark to the height of the airplane, and the slant distancebeing show to be 165.8 m.

FIG. 11 a: Illustration of operation of the present invention to draw aline mark connecting the base points of the two objects.

FIG. 11 b: Illustration of the present invention by showing in operationthe raising of the base points at the two ends of the line mark to theheight of their respective roof.

FIG. 12 a: Illustration of one particular aspect of the presentinvention, whereby the base points of the head point and the tail pointof the airplane are determined.

FIG. 12 b: Further illustration of the particular aspect of the presentinvention shown in FIG. 12 a: whereby the line connecting the two basepoints is shown.

FIGS. 13 a and 13 b: Further illustrate the present invention wherebythe heights of a multi-layered building are measured; whereby in FIG. 13a the height of the first layer of the roof is measured and locked, andthis height is 31 m, and whereby in FIG. 13 b the height of the secondlayer of the roof relative to the first layer is measured starting fromthe locked height.

FIG. 14: A schematic diagram of compensating the systematic biases ofthe object dimensions measured without using digital terrain models(DTM).

FIGS. 15 a, 15 b, 15 c, 15 d and 15 e: Illustrations of the baseselection method for 3D mapping in accordance with the presentinvention; whereby in FIG. 15 a the base point is selected, whereby inFIGS. 15 b and 15 c the Z level is changed via the user dynamicallyupdating the annotation and current Z level (as given by the yellow linein actual case); whereby in FIG. 15 d, the top of the building isoutlined using a 3D mapping tool (polygon); whereby in FIG. 15 e thebuilding footprint is enabled and displayed.

FIGS. 16 a, 16 b, and 16 c: Illustration of the roof-footprintdisplacement method for 3D modeling of flat roof buildings in accordancewith the present invention: whereby in FIG. 16 a, the mapping of theroof outline is shown using the 3D polygon mapping tool; whereby in FIG.16 b is shown the horizontal displacement of the roof outline tocoincide with the base of the building; whereby in FIG. 16 c is shownthat the 3D building model is constructed.

FIG. 17. A schematic diagram of roof shapes supported in modelingbuildings with gable, clipped gable, hip and shed roof shapes.

FIGS. 18 a, 18 b: Illustration of the creation of the 3D vector mapsfrom 2D vector maps in accordance with the present invention: whereby inFIG. 18 a, is shown the selection of 2D mapped vector after import intothe computer program; whereby in FIG. 18 b, is shown the 2D mappedvector mapped into a 3D vector by raising it to the top of the buildingthrough the use of the IGM.

FIG. 19: A system diagram generally illustrating the deployment of theinvention for web and network environment.

In the drawings, preferred embodiments of the invention are illustratedby way of examples. It is to be expressly understood that thedescription and drawings are only for the purpose of illustration and asan aid to understanding, and are not intended as a definition of thelimits of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 1. General Descriptionof the Invention

In one aspect of the present invention, a computer program is providedto obtain measurements of single, compound, multiple objects andobject-to-object relations from single imagery using measurement utilityand subsequently create 3D models and maps using model generationutility. 3D models and maps are conventionally performed using stereoimage pairs (generally referred to as “stereo pairs” in this context).The present invention is best understood, in one embodiment thereof, asa software product that is capable of obtaining 3D measurements andproducing 3D models and maps from the single images.

The interface of the computer program is best understood by reference toFIG. 1 a. Elements of the computer program of the present invention arebest understood in reference to FIG. 1 b. Utilities derived from theinvention allow for the development of 3D models and maps by usingmeasurements obtained from a single image.

The method of the present invention is best understood by reference toFIGS. 2 to 17, and is further described below. The computer program ofthe present invention consists of a computer application that is adaptedto provide instructions to a computer to implement the method of thepresent invention. The system of the present invention is bestunderstood by reference to FIG. 1 b.

In one aspect thereof, the present invention provides the measurementutility 8 designed to obtain 3D measurements of and between objects fromsingle imagery by using the projection, or the shadow, as well as theircombination (as shown in FIGS. 1 a and 1 b) and model generation utility9 designed to generate 3D models and maps.

-   -   This invention allows the collection of a wide range of        measurements as well as their derivatives (e.g., volume).    -   This invention uses the model generation utility 9 to quickly        and effectively construct 3D models (e.g., building) including        complex roof structures and subsequent 3D products (3D site maps        and 3D urban or natural scenes) all from single imagery.    -   As stated earlier, conventionally, 3D models are extracted by        using stereo image pairs. Often special viewing devices are        required to perform the extraction work. With this invention, 3D        measurements and 3D models can be obtained without using stereo        images or the special viewing devices.    -   With the use of RFM as the underlying IGM, the present invention        can be used for any sensor images (satellite and aerial, etc.)        with the RFM support and without changing any program        configurations. The present invention becomes scalable, flexible        and interoperable for any sensor imagery. That is, one program        can support multiple sensory images.    -   Moreover, measurements of many dynamic objects such as vehicles,        airplanes, clouds etc. including their moving features can be        obtained in accordance with the present invention, e.g., the        height and bearings of an airplane. Most airborne or satellite        based stereo pairs are not captured at the same time. Thus the        dimensions of the moving objects are not readily measured using        stereo pairs.

Measurements that are enabled by the present invention include: height,distance in 3D dimension, line of sight distance (e.g., distance betweentwo building roof tops, distance between the tower and the receiver),volume, bearings in 3D and their derivatives. 3D models and maps can begenerated using measurement information by using model generationutility. 3D models can be buildings, containers, tanks, towers etc. withcomplex roof structures.

The objects that can be measured in accordance with the presentinvention include:

-   -   A single object on the ground e.g., building, tower, tree etc or        ‘above’ the ground surface e.g., airplane, bridge, etc.    -   Compound objects: multi-layered complex building roofs, complex        constructions, etc.    -   Multiple objects: a cluster of objects, e.g., volume estimation        of a residential building block, damage assessment of forested        area    -   Object-to-object relationships: measurements relating to        object-to-object spatial relationships, e.g., the 3D distance        between a cellular tower and a receiver situated at a moving        ground vehicle.

The object can be either stationary (e.g., buildings, trees etc.) ordynamic (e.g., airplane, moving vehicle etc.). It also includes realobjects and synthetic objects (i.e., computer generated).

The present invention is capable of measuring, modeling and mappingobjects at various levels of details or ground sampling distances.Single imagery referred to in this disclosure may include:

-   -   Satellite images    -   Aerial images    -   Ground images, and    -   Other images acquired by sensors with an appropriately        calibrated image geometry model such as the RFM. The ground        sampling distances of these images can range from several inches        to several meters.

2. Description of the Interface

FIG. 1 a provides a conceptual drawing of a representative userinterface for accessing the functions of the measurement utility 8 andthe model generation utility 9 of the present invention. The userinterface is provided in a manner that is known.

-   -   Button 1 displays the image coordinates of the mouse in the        image plane.    -   Button 2 displays the ground coordinates of the object point        corresponding to the image point, and the ground coordinates are        computed using EQU. 1 as shown in FIG. 3. The datum and map        projection are preferably set in a dialog box.    -   Button 3 allows the input of image, IGM and DTM data (optional)        to the program    -   Button 4 allows the output of 3 D information including        dimensions, models and maps.    -   Button 5 turns the measurement utility 8 on/off. A dialog box in        the system gives two radio buttons for the selection of either        the projection measurement utility 12 (or projection ruler) or        the shadow measurement utility 14 (or shadow ruler) (these        utilities are illustrated in FIG. 1 b). Button 6 turns the model        generation utility 9 on/off. The computer program, in one        particular aspect thereof, implements a series of novel        algorithms for generating 3D models and maps, as particularized        below.    -   Button 7 displays the drawing results and associate information.

It should be understood that the present invention contemplates the useof alternate user interfaces for enabling an operator to access thefunctions of the measurement utility 8 and the model generation utility9.

3. Measurement Utility

As stated earlier, the present invention provides a measurement utility8 to measure the dimensions of and between objects from single imagerywith an IGM. The measurement utility 8 enables the processes of thepresent invention thereby permitting 3D measurement using the projectionor the shadow information or their combination. The measurement utility8 is best understood as a Windows™ program that enables the display ofsingle imagery (as described above) in accordance with its imagerygeometry, and processing of such single imagery in accordance with themethod of the present invention. Specifically, the measurement utility 8enables an operator to use a mouse or other suitable interface device topoint and click on selected points in the image, and thereby take themeasurements particularized below.

The measurement utility 8 is programmed in a manner that is known. Thedisclosure herein illustrates a representative computer applicationembodiment of the measurement utility 8 of the present invention. Asillustrated in FIG. 1 b, the measurement utility 8 is linked to adisplay facility 10 for displaying the images. The measurement utility 8is also linked to a data input source 11. This input data source 11stores all the data needed for the 3D measurements of objects, includingthe image data, the IGM and the DTM (optional). The data input foroperation of the measurement utility 8 is either stored to the database(or the file) (not shown), or in other embodiments, analyzed on the fly.The calculator 13 supports the functions of the projection measurementutility 12 and shadow measurement utility 14. The calculator 13 is bestunderstood as a utility that processes the input data in accordance withthe methods and equations described herein. The calculator also analyzesthe input data based on the model data (IGM or DTM, for example), andtherefore is also best understood as a modeler and/or analyzer as well.The model generation utility 9 is linked to the measurement utility 8.It implements algorithms that allow for the efficient reconstruction of3D models.

-   -   The present invention in one embodiment thereof relies on a        particular imaging process determined by the applicable IGM. The        imaging process generally provides the orientation information        of the imagery. In a particular embodiment of the invention, the        IGMs used is the rational function model, i.e. a sensor model        (OGC, 1999; Tao and Hu, 2001) that is capable of supporting        multiple sensors (sensor independence). The IGM used can also        include the well known models such as those based on        collinearity equations, direct linear transformation and others        etc.    -   The images used can be acquired by ground, airborne or satellite        platforms using different imaging sensors such as frame,        pushbroom or SAR etc.    -   The measurement utility 8 is preferably programmed such that it        can combine the projection and shadow of objects so as to        measure such objects (as particularized below).    -   The measurement utility 8 is also preferably programmed (as        stated earlier) to implement the processes particularized below        for measuring dynamic objects such as airplanes.    -   The measurement utility 8 can measure objects on the ground or        above the ground surface such as overpasses, bridges, viaducts        etc. The objects above the ground do not have physical base        points on the ground.

The measurement utility implements algorithms that are based onprojections or shadows of objects as well as both.

The conceptual explanation of the relationships between the elements ofthe measurable projection and shadow (including the base point 15, thetip point 16 and the shadow endpoint 17) is best understood by referenceto FIG. 2 a. FIG. 2 b shows the schematic diagram of the presentinvention about the relationship among the object, the IGM and the sunangles. An object is extruded vertically on the ground, and its heightis h. If the 3D coordinates of the base point are (X₀, Y₀, Z₀), the truetip of the object is at (X₀, Y₀, Z₂). Z₀ is the elevation of the bottomof an object, which can be retrieved from the digital terrain model(DTM) or a constant plane, or a value set by user. Z₂ is the elevationof the tip of an object.

The following equation is used to solve the (X, Y), as shown in FIG. 3.

$\begin{matrix}{\begin{bmatrix}v_{r} \\v_{c}\end{bmatrix} = {{\begin{bmatrix}\frac{\partial r}{\partial X} & \frac{\partial r}{\partial Y} \\\frac{\partial c}{\partial X} & \frac{\partial c}{\partial Y}\end{bmatrix}\begin{bmatrix}{\Delta\; X} \\{\Delta\; Y}\end{bmatrix}} - \begin{bmatrix}{r - \hat{r}} \\{c - \hat{c}}\end{bmatrix}}} & (1)\end{matrix}$where r and c are the row and column coordinates of the selected pointin the image; {circumflex over (r)} and ĉ are estimated values, and ΔXand ΔY are corrections.3.1 Projection Based Measurement Algorithm

The operator can obtain the height measurements by adjusting theelevation Z.

Example 1 Measuring when the Full Projection is Visible

An experiment was conducted to demonstrate advantages of the presentinvention in connection with projection-based measurement. A projectionruler can be drawn, visualizing adjusted height information iterativelyuntil the ruler line reaches the true tip of the object. In FIG. 4 a,line 1001 represents the outline of the building. As the operator beginsby indicating the base of the building (thick black circle 1002) andthen raises the height (thick black line 1003) of the floating cursor.As the cursor is raised iteratively, its position in the image iscomputed by the IGM on in the real time and the cursor is continuouslydrawn in the graphic user interface. Once the cursor touches the roofedge in the image (thick black circle 1004), this interactive procedurestops. In actual interface line 1003 will be appeared as green as arepresentative embodiment of the present invention. The height of theroof, as shown in FIG. 4 b is 48.6 m. This operation can be done at theboundary of the object's footprint.

3.2 Shadow Based Measurement Algorithm

Measurement on Flat Ground Surface

As shown in FIG. 2 b, the 3D coordinates of the shadow endpoint are (X₁,Y₁, Z₁), and Z₁ is equal to Z₀ for a flat ground surface. The shadowlength l of the shadow is determined by the sun's altitude. Therelationship among the length l, the object height h and the sunaltitude is determined by the following equations on the flat groundsurfaces:l=h/tan θ=(Z ₂ −Z ₀)/tan θ  (2)where h is the height of the object, θ is the sun's altitude.

In FIG. 2 b, assuming the terrain is flat, the coordinate offsets of theshadow endpoint relative to the object's position on ground, as shown inFIG. 5, are obtained by:

$\begin{matrix}{{{\Delta\; X} = {{X_{1} - X_{0}} = {{l \cdot {\sin(\alpha)}} = {{h \cdot \sin}\;{\alpha/\tan}\;\theta}}}}{{\Delta\; Y} = {{Y_{1} - Y_{0}} = {{l \cdot {\cos(\alpha)}} = {{h \cdot \cos}\;{\alpha/\tan}\;\theta}}}}} & (3)\end{matrix}$Measurement on Non-flat Ground Surface

The relationship among the shadow length l on the flat ground and theshadow length s on the slope with an angle of ψ, the object height h andthe sun altitude is determined by the following equations on non-flatground surfaces as shown in FIG. 6:

$\begin{matrix}{{{\Delta\; X} = {{X_{1} - X_{0}} = {{s \cdot {\cos(\psi)}}{\sin(\alpha)}}}}{{\Delta\; Y} = {{Y_{1} - Y_{0}} = {{s \cdot {\cos(\psi)}}{\cos(\alpha)}}}}{where}} & (4) \\{\psi = {\arctan\left( \frac{Z_{1} - Z_{0}}{\sqrt{{\Delta\; X^{2}} + {\Delta\; Y^{2}}}} \right)}} & (5) \\{s = {{{l \cdot \sin}\;{\theta/\sin}\;\left( {\theta + \psi} \right)} = {{h \cdot {{\cos(\theta)}/\sin}}\;\left( {\theta + \psi} \right)}}} & (6) \\{l = {{{h/\tan}\;\theta} = {{\left( {Z_{2} - Z_{0}} \right)/\tan}\;\theta}}} & (7)\end{matrix}$

Different cases of terrain relief are examined.

Steps of Shadow Based Measurement

FIG. 7 a illustrates the application of the method of the presentinvention to the measurable shadow of an object. This process flowgenerally has five steps. The operator selects the base point in theimage, whose ground coordinates are calculated using EQU. 1. Then, theoperator adjusts the value of the Z by the incremental change ΔZ. Theground coordinate offsets of the shadow endpoint are obtained using EQU.3 for flat ground, or EQU. 4 for non-flat ground at the vicinity of theobject. The shadow endpoint is cast in the image using theground-to-image transformation of the IGM, and the shadow ruler isplotted. The process is terminated if the shadow ruler fits the imageline well.

FIG. 7 b shows the process for the immeasurable shadow of an object.This workflow generally has six steps. The operator should select theshadow endpoint in the image, whose ground coordinates are alsocalculated using EQU. 1. Then, the operator adjusts the elevation Z bythe ΔZ. The computed offsets are subtracted from the endpoint toestimate the ground coordinates of the base point. Both the projectionruler and the shadow ruler are plotted. The projection ruler is used tojudge if it reaches the true tip of the object. The process isterminated if the both rulers fit the visible parts of the projectionand the shadow well in the image.

Example 2 Measuring when the Full Shadow is Visible

An experiment was conducted to demonstrate advantages of the presentinvention for the purpose of shadow-based measurement. A shadow ruler(in actual interface the line will be appeared as blue as arepresentative embodiment of the present invention) is drawn on theimage in the graphic user interface illustrate herein. Heightinformation is iteratively adjusted until the ruler line fits the wholeshadow in the image. As shown in FIG. 8 a, the operator begins bylocating the end point of the object's shadow (circle 1005) in the imageand then raises the height of the floating cursor. As the cursor israised, the position of the base point is updated as described in FIG. 7b, and their locations in the image are computed by the IGM. A line(dotted line 1006) connecting the base point and shadow endpoint and asecond line (1007) connecting the base point and the raised cursor aredrawn in the graphic user interface. Once the cursor reaches the topedge of the object in the image (circle 1008), this interactiveprocedure stops. The height of the roof, as shown in FIG. 8 b is 48.4 m,which is close to the height value measured using the projection utilityin Example 1.

Example 3 Measuring when the Projection and Shadow are Partially Visible

Several cases are performed to demonstrate advantages of the presentinvention of shadow-based measurement for immeasurable projections andshadows. In following cases, the base points of the objects can not belocated reliably or do not exist, but the shadow ruler can locate thebase point accurately. In FIG. 9 the shadow (1009) is measured by adotted line, and the projection (1010) is measured by a bold line. Theintersection of the two lines is the base point to be located.

FIG. 9 a shows the measurement of an airplane in the air. The airplaneis in the air and has no physical base point on the ground. The measuredheight is 57.1 m. In FIG. 9 b, an overpass is measured, the dotted line(1011) is the measured shadow length and the thick line (1012) is themeasured projected height (13.8 m). In FIG. 9 c, a tree is measured, thedotted line (1013) is the measured shadow length and the thick line(1014) is the measured projected height (29.4 m). FIG. 9 d shows themeasurement of a chimney whose base point can be located accurately, andthe height is 100.7 m. The dotted line (1015) is the measured shadowlength and the thick line (1016) is the measured projected height. Asshown in FIGS. 9 a to 9 d, the base points of these objects can beinferred from the shadow endpoints when using information about thesun's position and the IGMs and in actual cases a representativeembodiment of the these inventions of the measured shadow will beappeared as blue line and the measured projection will be appeared asgreen line.

Example 4 Measuring Object-to-Object Relations

An experiment was conducted to demonstrate advantages of the presentinvention for dimension measurement between any two points of twoobjects.

As shown in FIG. 10 a, the base point of the airplane on the ground isdetermined using the shadow ruler, and the height of the airplane is57.1 m (thick line 1017). Then a line mark (dash dot line 1018) is drawn(FIG. 10 b) to connect the base points and the landing points of therunway of the airport, and this distance on the ground is 158.1 m. Lastin FIG. 10 c, the base point of the line mark is raised to theairplane's height, and the slant distance (dash dot line 1019) becomes165.8 m.

As shown in FIG. 11 a, the slant distance (dash dot line 1019) is 193.1m when connecting the two base points of the two buildings. Both pointsare raised to their corresponding roof heights using the projectionruler, and the slant distance (dash dot line 1020) becomes 193.3 m inFIG. 11 b.

Example 5 Measuring the Bearing of Objects

An experiment was conducted to demonstrate advantages of the presentinvention of bearing measurement of any object. As shown in FIG. 12 a,the base points of the head point (at 1021) and the tail point (at 1022)of the airplane are determined using the shadow ruler. Then a bearingline mark (dash line 1023) is drawn to connect these two base points asshown in FIG. 12 b, and the angle is 285.2° under the UTM mapprojection.

Example 6 Measuring Compound Objects

An experiment was conducted to demonstrate the advantages of the presentinvention of height measurement of buildings with complex structures. Asshown in FIG. 13, the different levels of a multi-layered building roofcan be measured from single images using the projection and/or shadowrulers. In FIG. 13 a, the height (line 1024) of the first layer of theroof is measured using the projection ruler and then the height in thesystem is locked, and this height is 31 m. The height (line 1026) of thesecond layer of the roof relative to the first layer is measuredstarting from the locked height, and this height is 5.5 m as shown inFIG. 13 b. This shows that the height of the second layer is 36.5relative to the ground surface.

3.3 Compensation of Systematic Biases

When the measurements are performed with the absence of DTMs and/orGCPs, some systematic biases occur at both vertical and horizontaldirections. This results in changes in their dimensions and also makesthe positions of the objects measured displaced.

As shown in FIG. 14, the measurement error (Δh) of the object height isdetermined by the flying height (H), the object height (h), and thevertical shift (ΔH) due to the terrain availability as given by

$\begin{matrix}{{\Delta\; h} = {{h^{\prime} - h} = {{- \frac{\Delta\; H}{H}}h}}} & \left( {8a} \right)\end{matrix}$where h′ is the measured object height using the measurement utilities.Using EQU. 8a, the systematic errors of the object heights due to thevertical drifts can be compensated automatically for those objects whenDTMs become available later. Each object's height is correctedseparately since it usually has a different base height.

Similarly, the error of the horizontal dimensions of objects isdetermined by

$\begin{matrix}{{\Delta\; l} = {{l^{\prime} - l} = {\frac{\Delta\; H}{H}l}}} & \left( {8b} \right)\end{matrix}$where l′ and l are the measured and true object dimension, respectively.Using EQU. 8b, the systematic errors due to the vertical drifts can becompensated for automatically in the same manner as described above forthose object dimensions measured in 3D when DTMs become available later.

The corrections to the displacements of objects due to the absence ofDTMs can be accomplished by the calculation of that displacement for anypoint of the object. This process involves a few steps. First, both theraw image and the DTM are loaded into the computer system. Second, themeasured 3D object models are loaded. In one particular embodiment, amessage is popup to indicate if the bias compensation is needed. If so,a point (for instance, the first point) belong to the 3D models isprojected to the image plane using the IGM. Then the projected imagepoint is intersected with the DTM using EQU. 1. Third, the difference inthe X, Y, and Z coordinates between the original position of that pointand its position intersected with the DTM is calculated. Fourth, everypoint belongs to the 3D model is shifted by the same difference. Theupdated 3D models are preferably saved back to the disk files.

The corrections to the displacements of objects due to the absence ofGCPs can also be accomplished by carrying out a four-step procedure.First, the raw image, GCPs and optionally the DTM are loaded into thecomputer system, and the IGM is improved by using the GCPs. Second, apoint (for instance, the first point) belonging to the 3D models isprojected to the image plane using the original IGM. Then the projectedimage point is intersected with the DTM by using EQU. 1 and the improvedIGM. Third, the difference in the X, Y, and Z coordinates between theoriginal position of that point and its position intersected again iscalculated. Fourth, every point belonging to the 3D model is shifted bythe same difference. The updated 3D models are saved back to the diskfiles.

4. Collection of Measurements and Their Derivatives

Accordingly, as stated above the computer program of the presentinvention is operable to collect a wide range of 3D measurements of andbetween objects. It utilizes the objects' projection, shadow and theircombinations by using the measurement utility 8 as particularized above.

Many measurement derivatives can be developed after obtaining themeasurements in a manner that is known. These derivatives include butare not limited to volume, bearings, area, height difference, line ofsight, etc. Measurements and their derivatives can be used to generatemany 3D products for urban planning, landscape analysis, transportationand hydrological watershed analysis, emergency response, disastermitigation, and so on.

5. Creation of 3D Models and Maps

The computer program of the present invention creates 3D models and mapsby using the model generation utility.

Example 7 3D Model Creation Via Base Selection Method

In this approach to mapping a 3 D structure the Z level adjustment modeis enabled followed by marking the base of the structure using a mouseor similar pointing device (FIG. 15 a). Also, various key combinationscan be used to accommodate locking/unlocking the cursor. The Z level isthen adjusted for example using the page up/page down keys and thechange in the Z level (thick line 1027) is visualized to the user (inactual case by a yellow line) in the image plane (FIG. 15 b). When thedesired level is reached (FIG. 15 c) the user selects the 3D mappingtool, typically a polygon tool, and outlines the top (thick line 1028,in actual case by a pre-defined line color) of the structure as shown inFIG. 15 d. The Z level adjustment mode can be disabled and the basepoint of the projected structure can then be checked by enabling thedisplay footprint mode. The foot print mode uses the IGM to draw theprojected footprints (dotted line 1029, in actual case it is a darkcolor of the pre-defined outlined line color) as shown in FIG. 15 e. Thedesired 3D model can then saved into a database or file structure (notshown) if persistence storage is required. It can also be used togenerate a 3D virtual scene by capturing the visible faces of thebuilding and using them as textures in 3D visual models (not shown).

Example 8 3D Model Creation Via Roof-Footprint Displacement Method

The process of performing a 3D building modeling can be accomplished bycarrying out relative displacement or motion between the roof and thefootprint. This approach to mapping a 3D structure captures itsperimeter by first digitizing its roof outline (thick line 1030 as shownin FIG. 16 a) as projected in image plane. This outline is then shiftedin image plane (FIG. 16 b) to align with the base of the structure usingthe IGM. In the example provided, this is accomplished via pressing thepage up/page down keys (FIG. 16 b). This algorithm can create 3D models(FIG. 16 c) when part of the model footprint is not visible.

Many types of the building roof shapes are supported in thiscomputerized system. As shown in FIG. 17, some typical roof types areflat, gable, clipped gable, hip and shed roof shapes. The computerprogram can produce 3D building models with any complex roof types bythe combination of the basic roof types.

Example 9 Generation of 3D Maps from 2D Vectors

The 2D vector coordinates (r, c) are loaded into the computerapplication, and for each 2D position a Z level coordinate is assignedcoinciding with the base level of the mapped feature (FIG. 3). Eachfeature is then mapped into the 3rd dimension through the followingactions:

The user selects a feature (thick line 1031, usually denoted by 8circles) to be mapped (FIG. 18 a) into the 3^(rd) dimension and thenpresses the page up key to change the Z level coordinates of all nodesin the feature. Changes in the Z level are projected into image planevia the IGM giving visual feedback on the changing in Z level. When thedesired level is reached (FIG. 18 b) the user ceases changing the Zlevel and may opt to save the feature to the database (not shown).

6. Accuracy Assessment

The measurement accuracy has been extensively tested and evaluated. In asummary, these testing results suggest that sub-meter accuracies can beachieved, and are thus acceptable for a wide range of commercialapplications. The accuracy is dependent the flying height, objectheight, terrain availability, image resolution, image pixel measurement,and IGM accuracy.

The present invention also contemplates integration of additionalfeatures to enhance the operation of the present invention. For example,processes may be built into the functions of the measurement utilitythat enable, based on predetermined parameters, more efficientconvergence.

7. Computer Platform

The measurement utility and the model generation utility can be deployedin most popular computerized platform such as a PC, workstation, server,PDA, etc due to its simplicity to deploy, low overhead to computing,less restriction to the IGM (i.e., RFM) and no requirement for stereoviewing devices.

8. Web or Network Enabled Environment

The measurement utility 8 and the model generation utility 9 are idealfor network-based applications such as web, internet as well as wirelessnetworks given its simplicity to deploy, low overhead to computing, lessrestriction to the image geometry model and no requirement for stereoviewing devices.

FIG. 19 shows that the invention can be deployed in various forms(shaded component) in a web-enabled environment. It can be deployed as aclient application 20, web browser-based plug-ins (ActiveX controls) orJava applets 22, Application server 24 and Portal-based web service 26.The invention can also be embedded in wireless portals, PDAs orcell-phones etc. computerized platform with little modification, in amanner that is known.

9. Commercial Applications

This invention enables exploitation of the benefits of images for a widerange of applications including:

-   -   Obtaining critical facility and target information such as        building height, bridge clearance, road width, runway length, or        forest cuts;    -   Creation of 3D site maps of key facilities such as nuclear power        plants, airports, urban cities, critical infrastructures for        public safety and international intelligence;    -   Measuring area of damage (such as forest fire, flood,        earthquake) caused by disasters for insurance audits and        emergency response;    -   Modeling and planning for urban development, visualization and        simulation, gaming, government, transportation, civil        engineering etc.

Its applications are broad:

-   -   Defense    -   Environment    -   Homeland Security    -   Telecom    -   Visualization and Simulation    -   Agriculture    -   Local Government    -   Geology    -   Mapping    -   Forestry    -   Utilities    -   Real Estate    -   Transportation Planning    -   Insurance    -   Media    -   Entertainment and Gaming

Other variations and modifications of the invention are possible. Forexample, additional features can be built into the computer programproduct of the present invention to build on the basic 3D measurementand model data provided herein to provide for example density per squarekm, urban shadow estimation, etc based on single imagery. The computerproduct of the present invention can be integrated with otherapplications. All such modifications or variations are believed to bewithin the sphere and scope of the invention as defined by the claimsappended hereto.

1. A system for deriving three-dimensional coordinate data comprising: acomputer which accepts a two-dimensional single image of an object,wherein the single image includes image data and is associated with animage geometry model (IGM); an analyzer that derives at least one set ofthree-dimensional metadata from the single image; a measurement utilitythat measures dimensions of the object by performing a projectionmeasurement and a shadow measurement for the object, wherein theprojection measurement is based at least in part on one or more of anIGM, height distance of the object, point-to-point distance of theobject, line-of-sight distance of the object, volume, and bearing, andfurther wherein performing said shadow measurement includes determininga length of the shadow of the object based on a height of the object andan altitude of the sun; and a model generation utility that derivesthree-dimensional coordinate data based at least in part on theprojection, the shadow, and the three-dimensional metadata.
 2. Thesystem of claim 1, wherein the IGM is a geometric relationship between aspace of the object and a space of the single image.
 3. The system ofclaim 2, wherein the shadow measurement is based at least in part on anIGM, height, point-to-point distance of the objects, the objects may bestatic or dynamic.
 4. The system of claim 2, wherein the IGM utilizes atleast one of a physical sensor model, a generalized sensor model, arational function model (RPM), collinearity equations, direct lineartransformations, or any combination thereof.
 5. The system of claim 1,wherein the object is a single static object, a compound object,contains multiple sub-objects, stationary, dynamic, real, synthetic, orany combination thereof.
 6. The system of claim 1, wherein theprojection measurement determines an elevation for the object based atleast in part on a digital terrain model (DTM).
 7. The system of claim1, further comprising an interface component that displays contentincluding at least one of the two-dimensional single image, the object,the three-dimensional metadata, the projection, the shadow, the objectspace, the image space, or any combination thereof, the interfacecomponent further accepts an input from a user that permits the user tointeract with and manipulate the displayed contents.
 8. The system ofclaim 1, wherein the analyzer compensates for systematic biasesoccurring in horizontal and vertical directions when capturing thesingle images.
 9. A non-transitory computer software product comprisinga computer program that, when executed on a computer, causes thecomputer to perform a method for creating three-dimensional models, themethod comprising: receiving at least one two-dimensional single imagefor an object, wherein the single image includes image data, and furtherwherein the single image is associated with an image geometry model(IGM); deriving at least one set of three-dimensional metadata from atleast one of the single images; measuring a projection and a shadowmeasurement for at least one of the objects, wherein said shadowmeasurement is based, at least in part, on an altitude of the sun; andgenerating three-dimensional coordinate data based at least in part onthe projection, the shadow and the three-dimensional metadata.
 10. Thesoftware product of claim 9, wherein the IGM is a geometric relationshipbetween a space of the object and a space of the single image.
 11. Thesoftware product of claim 9, further comprising measuring a derivativeof the projection and a derivative of the shadow.
 12. The softwareproduct of claim 9, wherein one of the single images is at least one ofa satellite image, an aerial image, a ground image, or an image based ona rational function model (RPM).
 13. The software product of claim 9,wherein measuring the projection and the shadow includes iterativeadjustments of locating an end point of the shadow of the object,updating a base point position, and connecting the base point and theshadow endpoint.
 14. The software product of claim 9, further comprisingcompensating for systematic biases in the projection measurement andshadow measurement.
 15. The software product of claim 9, wherein theprojection measurement determines an elevation for the object based atleast in part on a digital terrain model (DTM).
 16. The software productof claim 9, wherein measuring the shadow includes selecting a base pointin the image, calculating a set of ground coordinates of the base point,adjusting the height, calculating at least one offset of an estimatedshadow endpoint relative to the base point, and casting the shadowendpoint to the image.
 17. The software product of claim 9, thethree-dimensional metadata includes an object geometry.
 18. The softwareproduct of claim 9, wherein receiving the at least one two-dimensionalsingle image includes acquiring the image utilizing at least one of aframe sensor, pushbroom sensor, or any combination thereof.
 19. A methodfor deriving three-dimensional measurement information comprising: meansfor receiving at least one two-dimensional single image for an object,wherein the at least one two-dimensional single image includes imagedata and is associated with an image geometry model; means for derivingat least one set of three-dimensional metadata from at least one of thesingle images; means for measuring a projection and a shadow measurementfor at least one of the objects, wherein said means for measuring ashadow measurement include a computer that executes a computer program,wherein the computer program causes the computer to determine a lengthof the shadow of the at least one of the objects based, at least inpart, on an altitude of the sun; and means for generatingthree-dimensional coordinate data based at least in part on theprojection, the shadow, and the three-dimensional metadata.